i wy Treasury Board of Canada Secrétariat du Conseil du Trésor 
u da 


Secretariat 


Better government: with partners, for Canadians 


a 


> Theory-Based 
Approaches 

to Evaluation: 
Concepts and Practices 


Canada 


© Her Majesty the Queen in Right of Canada, 
represented by the President of the Treasury Board, 2012 


Catalogue No. BT22-131/2012E-PDF 
ISBN 978-1-100-20593-9 


This document is available on the Treasury Board of Canada Secretariat 
website at http://www.tbs-sct.gc.ca 


This document is available in alternative formats upon request. 


Table of Contents 


Prefa Ce ss nn ent ane tn el OED E ALEE er ne er ee iii 
Acknowledgements anses pre ae ee ee eee Hee ee as re iv 

EPO CUCTION sis nist orne dees ta ee pe PR A Net er nier cid 1 
2. Context and Causation in Theory-Based Approaches......................... 4 
3. Theories of Change and Logic Models ...............:ccceeeee eee eee eee ee teeta eee 5 
4. Theory-Based Approaches to Evaluation ..............:ccceceeee cece ereere 7 
5. Addressing Federal Core Issues Using Theory-Based Approaches 

LG: EMAIHO LION Re ARR atin th a ee ee ue En ee eae 11 
6. Developing Theories of Change... 13 

Using Theory-Based Approaches to Make Causal Inferences............ 20 
8. Strengths and Weaknesses of Theory-Based Approaches 

HO EVO a 525th ve osc eas va ea let ahs eee tee eae ata idee 25 
9. Using Theory-Based Approaches to Evaluate Cause- Effect Issues 

in Different Types of Interventions... 27 
10. Other Uses of Theory-Based Approaches....................................... 33 
LL: ‘From Concept to Practice icii nasale acne RPM andres re 35 
12. For More LniOrmMmation:sss sienne cs erential Matte ed Nr ete 36 
ERIONGEGS RS Rd le AA SEA 37 


Theory-Based Approaches to Evaluation: Concepts and Practices 


Preface 
Why Theory-Based Approaches? 


Over the last 20 years, theory-based approaches — under various names — have increasingly 
moved into the mainstream of thinking and practice about how interventions (i.e., programs, 
policies, initiatives or projects) are designed, described, measured and evaluated. During that 
time, theory-based approaches have demonstrated promise in helping evaluators address a 
variety of challenges, such as coming to terms with the inherent complexity of certain types of 
interventions and overcoming the limitations of experimental evaluation designs. 


To help ensure that evaluation continues to play a key role in providing a neutral, evidence-based 
assessment of the value for money (i.e., relevance and performance) of federal government 
programs, the Treasury Board of Canada Secretariat recognizes the importance of expanding the 
“tool box” available to federal evaluators. Theory-based approaches to evaluation represent a 
potentially powerful tool. 


Using This Document 


This document introduces some of the key concepts of theory-based approaches to evaluation. It 
is hoped that readers will be encouraged by the information and advice provided in this 
document and will explore the use (e.g., through pilot evaluations) of theory-based approaches to 
evaluation in a federal setting. To support this, Sections 1.0 to 8.0 of the document describe the 
general application of theory-based approaches to evaluation, and Sections 9.0 and 10.0 discuss 
the potential application of theory-based approaches to a range of federal programs. 


This document is neither an exhaustive training program in theory-based evaluation nor a 
step-by-step guide to undertaking a theory-based evaluation. Evaluators who wish to integrate 
theory-based approaches into their practices are encouraged to pursue additional readings 
(including those referenced in this document) and, as appropriate, to seek additional support in 
undertaking a theory-based evaluation. 
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1. Introduction 


1.1 The Challenge of Experimental Evaluation Designs 


As noted in Program evaluation methods: Measurement and attribution of program results 
(Treasury Board of Canada Secretariat, chap. 3), evaluators face two broad challenges: 
measuring the expected results’ from an intervention and attributing those results to the activities 
of the intervention. 


Experimental evaluation designs aim to address both of these challenges (chap. 3). These designs 
typically measure both the baseline and the final results associated with an intervention and, by 
incorporating a counterfactual (e.g., a comparison group), can assess the causal link between the 
intervention and the observed results. 


Experimental and quasi-experimental evaluation designs can be quite powerful and should be 
undertaken when appropriate (e.g., when interventions involve a new or different approach to 
addressing a problem and the goal of the evaluation is to test whether the intervention works). 
However, there are several shortcomings associated with these designs, in particular: 


> Practicality: In many contexts, experimental designs, especially the more sophisticated ones, 
cannot be implemented. Often the development of a counterfactual may be difficult or 
undesirable (e.g., for ethical reasons). In some cases, there may not be an opportunity to 
manipulate the delivery of the intervention as required in order to demonstrate attribution. In 
other cases, the resources or time required to undertake experimental designs may not be 
available. 


> Seeing interventions as black boxes: Experimental designs, even when feasible, are not 

aimed at understanding why and how the observed results occurred. These designs do not 
attempt to answer questions such as the following: “What was it about the intervention or the 
context that caused the results? Where the expected results were not observed, what was it 
about the intervention that didn’t work? Was the underlying theory of the intervention wrong, 
or was the problem a case of poor implementation?” Knowing the answer to these questions 
can be valuable for improving the intervention or implementing the intervention in a different 
location or manner. Because experimental designs do not ask these questions, they are often 
described as “black box studies”; they may assess whether the expected results occurred and 


1. In this context, result refers broadly to the collection of outputs and outcomes associated with an intervention. An 
intervention is a project, program, policy or initiative. Expected results are those results that an intervention 
seeks to achieve. Observed results are the results that have demonstrably or measurably occurred (whether 
expected or unexpected). 
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whether the intervention played a role, but they do not explore the reasons why the 
intervention did or did not work. 


A theory-based approach to evaluation can help address these shortcomings. In the absence of an 
overall experimental design, it provides a way to assess the extent to which an intervention has 
produced or influenced observed results. It also opens the black box, examining what role the 
intervention played in producing the observed results. 


1.2 What are Theory-Based Approaches to Evaluation? 


Theory-based approaches to evaluation use an explicit theory 
of change to draw conclusions about whether and how an Theory-based evaluation is an 
. i : £ approach to evaluation (i.e., a 
intervention ones to one oo Theory-based conceptual analytical model) 
approaches are a “logic of enquiry,” which complement and and not a specific method or 


can be used in combination with most of the evaluation technique. It is a way of 
structuring and undertaking 


designs and data collection techniques outlined in Program analysis in an evaluation. 


evaluation methods: Measurement and attribution of program 
results." 


A theory of change explains how an intervention is expected 


: ; | Theories of change can be 
to produce its results. The theory typically starts out with a a 


thought of as the story of what 
sequence of events and results (outputs, immediate outcomes, should happen in the “arrows” 


intermediate outcomes and ultimate outcomes) that are that link the boxes in a 


| . : oe traditional logic model. 
expected to occur owing to the intervention. This is commonly 


referred to as the “program logic” or “logic model.” However, | Another way to think of a theory 
he th fch further b lini h hani of change is as a logic model 
the theory of change goes further by outlining the mechanisms | that has been described and 
of change, as well as the assumptions, risks and context that explained, in particular in terms 
support or hinder the theory from being manifested as of a ne wag E k 

z outputs an e different levels 
observed outcomes. This opens the black box of change and of oütcöme. 
allows evaluators to better examine the causal link between 
the intervention outputs and the observed outcomes. The theory of change can be used to test — 
with evidence — the assumed causal chain of results with what is observed to have happened, 


checking each link and assumption in the process to verify the expected theory. 


Theory-based approaches have been discussed in evaluation literature for many years (Weiss, 
1997; Rogers, 2007; Funnell & Rogers, 2011). While there is little agreement either on the 
terminology or the concepts, there is consistency in the main messages and agreement on the 
value of theory-based approaches. 
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2. Context and Causation in Theory-Based Approaches 


Two key ideas that distinguish theory-based approaches from traditional approaches are (1) the 
influence of context on program results, and (2) a mechanistic, rather than counterfactual, 
approach to determining causality. 


Context Matters 


Theory-based approaches, more than many other evaluation When developing a theory of 
approaches, pay explicit attention to the context of the change, the context may be 
intervention. It is acknowledged that contextual factors can explicitly identified (i.e., as a 


i ; : ; eee i stand-alone description) and/or may 
help an intervention achieve its objectives or act against the be represented through a 


intervention working. For example, in an intervention aimed at | discussion of the assumptions 
underlying the program. 


reducing smoking in the population, contextual factors may 
include the enthusiasm of those required to enforce a ban, the social makeup of the targeted 
populations, and related supporting legislation. These factors are often essential in making causal 
inferences and need to be part of the evaluation design. 


Mechanistic Causation 


For most interventions, there are usually multiple causes for an observed outcome. A wide range 
of other economic and social factors, not to mention other government interventions, may also 
come into play. For example, the success of an allowance-based incentive program intended to 
encourage nurses to live and work in rural areas may be influenced not only by the existence and 
size of the allowance but also by employment rates in the nursing field, the specific rural settings 
in which nurses are placed, and the personal backgrounds of the nurses involved. In these 
situations, seeking a clear “one-to-one” causation that can be wholly attributed to one mechanism 
(finding the cause) is not possible. Rather, the relevant evaluation question is: In light of the 
multiple factors influencing a result, has the intervention made a noticeable contribution to an 
observed outcome and in what way? Understanding contribution, rather than proving attribution, 
becomes the goal. 


Theory-based approaches to evaluation attempt to understand an intervention’s contribution to 
observed results through a mechanistic or process interpretation of causation, rather than 
determining causation through comparison to a counterfactual. In theory-based approaches, the 
specific steps in a causal chain, the specific causal mechanisms, are tested. If these can be 
validated by empirical evidence, then there is a basis for making a causal inference. At the same 
time, theory-based approaches seek to identify and assess any significant influencing factors (i.e., 
contextual factors) that may also play a role in the causal chain and thus affect the contribution 
claim. 
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3. Theories of Change and Logic Models 


Managers of federal programs 
regularly use logic models to 
describe the results expected from 
an intervention.” " Where program 
managers have not developed their 
own logic model, evaluators often 
do so to support their evaluation 
effort. The results chains 
embedded in logic models are key 
building blocks for developing 
theories of change. Theories of 
change expand on results chains to 
articulate why the sequence of 
results is expected to occur, 
whereas logic models tend to focus 
solely on the results intended by a 
program (i.e., on the boxes ina 
typical visual logic model). The 


Figure 1. Short-Cycle Logic versus Long-Cycle Logic 
(Theory of Change versus Program Logic) 


Results Chain 


Ultimate 
Outcomes 


Intermediate 
Outcomes 
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(Theory of Change) 


Long-Cycle Logic 
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© 


Activities 


theory-based approach argues that the “logic of the logic” is the important feature of logic 
models; it focuses on the connections (which can be thought of as the “short-cycle” logic) 
between the boxes in a visual logic model rather than the “long-cycle” logic of the results chain 
(see Figure 1). Simply put, theories of change explain how the intervention is expected to bring 
about the desired results rather than just describing the results. 


Generally, a theory of change includes: 


> a logic model/results chain; 


> the assumptions, risks and, in some cases, the 


Theories of change are referred to by a 
variety of names, including program theories, 


mechanisms associated with each link in the impacts pathways, and pathways of change. 


logic model/results chain; 


> the external factors that may influence the expected results; and 


2. Under the Policy on Evaluation (2009), deputy heads of federal departments and agencies must ensure that 
performance measurement in support of evaluation is implemented for all programs. The plans for performance 
measurement, including logic models, are to be outlined in a document referred to as a Performance 
Measurement Strategy. For more information on performance measurement strategies, consult Supporting 
Effective Evaluations: A Guide to Developing Performance Measurement Strategies on the Centre of Excellence 


for Evaluation website. 
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> any empirical evidence supporting the assumptions, risks and external factors. 


Assumptions are key events or conditions that must occur for the causal link to happen. Risks are 
influences or events outside the intervention that may inhibit the causal link from happening. 
Mechanisms are the causal processes that enable the program to produce results. External factors 
are circumstances beyond the control of the program, such as social, political or economic 
context, which may affect the program’s ability to achieve an intended result. 


In some cases, theories of change are subdivided into two components: the intervention theory, 
which outlines the underlying behavioural assumptions (the mechanisms) behind the 
intervention, and the implementation theory, which identifies how an intervention is expected to 
operate and trigger these mechanisms. These components can be developed separately, but are 
often merged into or developed as one theory of change. Unfortunately, there is no consistent 
terminology for theories of change; different authors may use the same term for different 
concepts. Blamey and Mackenzie (2007) discuss this issue. 


References 


Blamey, A., & Mackenzie, M. (2007). Theories of change and realistic evaluation: Peas in a pod 
or apples and oranges. Evaluation, 13(4), 439-455. 
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4. Theory-Based Approaches to Evaluation 


There is no agreed classification of theory-based approaches; indeed, in recent years, there has 
been a proliferation of theory-based approaches and numerous variations within each approach. 
In this section, two prominent categories of theory-based evaluations, realistic evaluation and 
theory-of-change approaches, are discussed. These descriptions are generic and may not always 
apply. For more information on the similarities and differences in theory-based approaches, 
readers can consult Blamey and Mackenzie (2007) or Stame (2004). 


Realistic Evaluation 


Realistic evaluation is a form of theory-based evaluation developed by Pawson and Tilley (1997, 
2006). They argue that whether interventions work depends on the underlying mechanisms at 
play in a specific context. For Pawson and Tilley, 


outcome = mechanism + context 


Mechanisms describe what it is about the intervention that triggers change to occur. In a smoking 
cessation intervention, for example, mechanisms might include peer pressure to stop or to not 
stop, fear of health risks, and economic considerations. For realistic evaluators, the key 
evaluation questions are, What works? For whom? In what circumstances? In what respects? 
How? Realistic evaluators are less interested in the outcome-level question, Did the intervention 
work at a macro level? 


Realistic evaluation develops and then empirically tests the hypotheses about what outcomes are 
produced by what mechanisms in what contexts. The realistic approach tends to be more 
research-oriented, focusing on the underlying intervention theory and its behavioural 
assumptions at work, and the conditions supporting the intervention. The focus is on the most 
promising context-mechanism-outcome configurations (CMOCs), which show how interventions 
are meant to work in which populations and under what conditions. These can be viewed as 
mini-theories of change or links in an overall theory of change of an intervention. Each CMOC 
is, in effect, the subject of an evaluation and is tested against the available evidence. 


Blamey and Mackenzie (2007, p. 444) describe how to undertake a realistic evaluation, using a 
smoking cessation intervention as an example: 


Step 1: The evaluator, through dialogue with program implementers, attempts to 
understand the nature of the program: What is the aim of our smoking cessation program? 
What is the nature of the target population at whom it is aimed? In what contexts and 
settings will it operate? What are the prevailing theories about why smoking cessation 
services will work for some people in some circumstances? 
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Step 2: The evaluator maps out a series of potential mini-theories that relate the various 
contexts of a program to the multiple mechanisms by which it might operate to produce 
different outcomes. For example, practitioner knowledge and the existing evidence might 
suggest that focusing the educational component of a midwife-led smoking cessation 
program on the potential negative effects on babies in utero will be most effective for 
pregnant women who have no children. However, young, non-pregnant female smokers 
may be less likely to respond to concerns about the threat of health effects on 
non-existent babies, but may be more likely to respond to anti-smoking interventions 
designed to appeal to their self-image. 


Step 3: At this stage, the evaluator undertakes an outcome inquiry in relation to these 
mini-theories. This involves developing a quantitative and qualitative picture of the 
program in action. It might, for example, address how different types of smokers fare 
when it comes to breaking the habit following different types of cessation services 
delivered in a variety of ways. This picture includes an assessment of the extent to which 
different underlying psychological motivations and mechanisms have been triggered in 
particular smokers by specific services. 


Step 4: By exploring how CMOCs play out within a program, the evaluator refines and 
develops tentative theories of what works for whom in what circumstances. 


A key strength of the realistic approach is its focus on context. Context must be part of the 
evaluation framework, and specific contexts, whether within or outside the control of the 
intervention, can enhance or detract from how well the intervention works. 


Examples of realistic evaluations are found in Byng, Norman and Redfern (2005); Leeuw, Gilse 
and Kreft (1999); and Leone (2008). 


Theory of Change Approaches 


These approaches involve developing a theory of change for the intervention showing how the 
specific intervention is intended to work and the assumptions behind the theory. They tend to 
address the traditional evaluation questions of whether and to what extent the intervention has 
worked (i.e., has made a difference to the desired outcome). The theory of change is usually 
developed on the basis of a range of stakeholders’ views and information sources. Approaches 
include theory-based evaluation (Weiss, 1995, 2000), theory-driven evaluation (Chen, 1990), and 
contribution analysis (Mayne, 2001). All develop a theory of change for the intervention and 
then verify the extent to which the theory matches what is observed. 
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One theory of change approach, contribution analysis, argues that if an evaluator can validate a 
theory of change with empirical evidence and account for major external influencing factors, 
then it is reasonable to conclude that the intervention has made a difference. The theory of 
change provides the basis for arguing that the intervention is making a difference and identifies 
weaknesses in the argument, thus identifying where evidence for strengthening such claims is 
most needed. Causality is inferred from the following evidence: 


> The intervention is based on a reasoned theory of change: the results chain and the underlying 
assumptions of why the intervention is expected to work are sound, plausible, and agreed to 
by key players. 

> The activities of the intervention were implemented. 


> The theory of change is verified by evidence: The chain of expected results occurred, the 
assumptions held, and the (final) outcomes were observed. 


> External factors (context) influencing the intervention were assessed and shown not to have 
made a significant contribution, or if they did, their relative contribution was recognized. 

In the end, a conclusion (a contribution claim) is made about whether the intervention made a 

difference. To summarize: 


contribution claim = verified theory of change + other key influencing factors accounted for 


Examples of theories of change evaluations are found in Carvalho and White (2004), Weiss 
(1995), and White (2009). Patton (2008) provides an example of an evaluation using contribution 
analysis. 
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5. Addressing Federal Core Issues Using Theory-Based 
Approaches to Evaluation 


The Treasury Board Directive on the Evaluation Function” identifies five core issues — three 
related to program relevance and two related to program performance — that should be 
addressed in all federal evaluations undertaken in response to the Policy on Evaluation’: 


Relevance 


1. Continued Need for the Program 
2. Alignment with Government Priorities 
3. Alignment with Federal Roles and Responsibilities 


Performance (Effectiveness, Efficiency and Economy) 


4. Achievement of Expected Outcomes 
5. Demonstration of Efficiency and Economy 


Theory-based approaches can be particularly useful in addressing issues one, four and five. 


Continued Need for the Program is an assessment of the extent to which the intervention 
continues to address a demonstrable need and is responsive to the needs of Canadians. The 
development and assessment of the intervention’s theory of change can assist in examining, in 
light of current knowledge about an intervention of this type, whether the structure and design of 
the current intervention is still appropriate. Does it still make sense? Is the current design right? 
Is the program targeting the right people in the right ways? Also, a theory of change that includes 
details on the original (or evolving) programming context can be compared with the current 
context to help determine what changes may be affecting program performance and/or the degree 
to which the program has been able to adapt to those changes. 


Achievement of Expected Outcomes involves an assessment of progress toward expected 
outcomes, including immediate, intermediate and ultimate outcomes, with reference to 
performance targets, intervention reach and intervention design, including the linkage and 
contribution of outputs to outcomes. Theory-based approaches can help evaluators to draw 
conclusions about the contribution made by an intervention to the observed outcomes. 
Furthermore, theory-based approaches can provide information about which aspects of the 
intervention worked well and in what circumstances, and on what other factors affected the 
outcomes of interest. 
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Demonstration of Efficiency and Economy involves an assessment of resource utilization 
relative to the production of outputs and progress toward expected outcomes. Theory-based 
approaches involve the isolation of results chains and the detailing of mechanisms and processes 
by which interventions are intended to operate. In evaluations involving an operational approach 
to assessment of core issue five (e.g., formative or process evaluations), these program or 
intervention theories can help identify units of analysis for the assessment of resource utilization. 
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6. Developing Theories of Change 


There is considerable literature (see References) on developing theories of change. Much of it is 
written for and by evaluators, who often have to develop such theories when they undertake 
evaluations. A preferred and increasingly common scenario is for intervention designers and 
managers to develop a theory of change during the initial design of the intervention as an aid to 
intervention planning and planning for performance measurement. This is good practice for 
results-based management and is encouraged where appropriate. Section 10.0 discusses these 
uses of theories of change. 


When evaluators are retroactively developing a theory of change, the following sources of 
information should, at a minimum, be consulted: 


> Key intervention documents (e.g., Memoranda to Cabinet, Intervention Terms and 
Conditions, and planning documents) 


> Relevant literature review (e.g., prior evaluations and social science research) 
> Intervention managers 

> Beneficiaries 

> Subject-matter experts 


> The program’s logic model 


The development of a theory of change generally involves three steps: 
1. Developing a logic model with clear results chains and explicit causal links (a basic theory of 
change) 


2. Identifying assumptions and risks underlying the theory of change 


3. Identifying other contextual factors associated with the results chain. The result is a refined 
theory of change. 


Developing a Basic Theory of Change 


The development of a basic theory of change involves identifying an intervention’s activities, 
outputs, and the sequence of outcomes needed for the expected results to occur. This often 
involves developing a logic model with reasonable results chains and describing what the arrows 
or other connections in the results chains imply. 


For example, Figure 2 provides an example of a basic theory of change for an intervention aimed 
at building effective results-based management (RBM) in an organization. The left-hand side of 
Figure 2 shows a results chain for the intervention, which includes providing training and 
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guidance on measuring and monitoring results, which leads to improved systems, capacity and 
institutionalization of RBM, which leads to the use of results information to inform decision 
making; which leads, ultimately, to the improvements in organizational program performance. 
The right-hand side of Figure 2 identifies the causal links between the outputs and outcomes (i.e., 
what “happens” in the gray arrows that link the outputs and outcomes). An outline or summary 
of the intervention’s basic theory of change appears at the bottom of Figure 2. 


The blue arrows, which flow backward from the ultimate outcome through to outputs, indicate 
that although the results chain is pictured as a linear process, there is normally feedback between 
the different stages in the results chain. For example, the availability of more and better results 
information can lead to the need for more training and guidance. Similarly, an increase in the use 
of results information can lead to an increase in demand for good results information. 


In developing theories of change, it is also useful to identify the degree of influence the 
intervention has in terms of its causal link, the degree of control the intervention has in ensuring 
that the causal link is realized. Figure 2 illustrates this by labelling the causal links as being 
directly influenced [DI], influenced [I] or outside the influence [O] of the intervention. 
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Figure 2. À Basic Theory of Change for Enhancing 
Results-Based Management in Organizations 


Results Chain Causal Links 


Ultimate 
Outcomes 


Causal Link: Regular use of results information 
leads to more effective, efficient and relevant programs 
and to more credible accountability reporting. [0] 


Causal Link: Over time, the RBM frameworks 

and systems show their worth and the organization 
Intermediate integrates and refines them into how they 
Outcomes do business. [I] 


Causal Link: As a result of regular, credible results 
information being available, the organization values 
and uses evidence in planning, implementing and 
reporting on its performance. [|] 


Immediate 
Outcomes 


Causal Link: RBM capacity across the organization 
is enhanced and corresponding RBM systems are 
developed as a result of information and training 


provided [DI]; Professional RBM capacity is 
acquired [DI]. 


Outputs 


The outline of the intervention theory of change: By providing information, training and hands-on 
assistance in RBM, organizations will build up their RBM capabilities and related systems and use the 
results information to inform their decision making, leading to more effective and efficient programs. 
Legend: [C] control, [DI] direct influence, [I] influence, [0] outside of influence 
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Refined Theories of Change: Identifying Assumptions, Risks and External Factors 


Refined theories of change go beyond the basic theory to identify the assumptions behind the 
various causal links in the results chain and the risks associated with those assumptions. This 
helps ensure that the theory of change explains what conditions have to exist for each causal link 
to be realized (i.e., for A to lead to B). Figure 3 sets out a refined theory of change for the 

RBM example, showing the assumptions and risks behind the causal links identified in the basic 
theory of change. Alternatively, instead of representing the assumptions as done in Figure 3, they 
can be presented as a list of premises that need to be tested. This is the approach that Leeuw, 
Gilse and Kreft (1999) used. 


As with the basic theory of change, it is useful in refined theories of change to identify the 
degree of influence that an intervention has over these assumptions and risks. However, as noted 
in Figure 3, a new level of influence (control [C]) is introduced to denote areas where the 
intervention should be able to effectively control a particular condition (e.g., the production of 
outputs). 


It is also important to identify the significant external factors, or contextual factors, that might 
influence the intended outcomes. These external factors are generally situations or events that are 
outside the direct control of the intervention to influence, manage or prevent. These can be 
illustrated as part of the theory of change. In Figure 3, for example, external influences, 
identified on the left-hand side of the results-chain, may include requirements of funding 
agencies, negative experiences with past management initiatives or management trends among 
peers to improve intervention monitoring and evaluation. 


Setting out the assumptions, risks and external influences helps describe both the intervention 
and the context in which it is operating. Defining assumptions, risks and external factors at the 
intervention design stage can help identify additional activities that the intervention may wish to 
undertake as part of its risk management. This may allow an identified external influencing 
factor over which the intervention previously had no influence to be converted to an internal risk 
that can, to some extent, be mitigated. This expanded results chain/theory of change can be 
useful as a framework both for evaluation purposes (where the theory of change would be tested 
with empirical data from the intervention to determine whether the theory is working) and for 
enhancing reporting on the performance of the intervention (see Section 10). 


Discussions on developing theories of change and examples are available on the Theory of 
Change Community website.” 
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Figure 3. À Refined Theory of Change for Enhancing Results-Based 
Management in Organizations 


Results Chain Causal Links 


Assumptions: Programs are competently delivered [0]. 
Ultimate Risks: Delivery partners do not appreciate the value 
Outcomes of results information [1]. Changes are made to 
organizational funding [0]. Frequent changes occur 
in priorities [0]. 


Intermediate 
Outcomes 


Assumptions: Senior management visibly supports 
using results information [I]. A robust results culture is 
established leading to informed demand for results 
information [I]. Results information is timely, pertinent 
and credible, and not overwhelming [I]. RBM regime is 
kept fresh and relevant with regular review so as not to 
become a burden [1]. 

Risks: Frequent changes in strategic priorities render 
results information less relevant. [0]. 


Assumptions: Trial RBM efforts demonstrate their 
worth [I]. Useful and credible results information is 
produced [1]. Senior management support continues [1]. 


Measurement systems are practical and credible [DI]. 
Systems are user-friendly [1]. 

Risks: RBM initiative is sidelined by other events [0]. 
The RBM intervention seems at odds with other 
initiatives [I]. 


Assumptions: Appropriate resources are forthcoming 
[I]. RBM frameworks and systems are seen as useful 


— [DI]. Enhanced capacity is adequate to support further 
RBM development [DI]. 
Risks: Negative peer experiences undermine 
support for RBM [I]. Organizational culture 
is difficult to change [I]. Funding agency 
expectations are high [I]. 
Outputs 


Assumptions: Appropriate audiences in the 
organization are reached [C]. Training and information 


is effective [DI]. Adequate acquisition of in-house 
professional RBM capacity takes place [DI]. 


Legend: [C] control, [DI] direct influence, [I] influence, [0] outside of influence 
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7. Using Theory-Based Approaches to Make Causal 


Inferences 


Causal inferences (claims about causes and effects) can be made by testing the theory of change 
for an intervention against what has been observed, and assessing the influence of other external 
factors. One such approach discussed in this section is contribution analysis. 


Figure 4 sets out seven iterative steps in a contribution analysis. Each step in this process adds to 
the contribution claim and helps address the weaknesses identified at the previous step. The 
result of a contribution analysis should be a reasonably credible “contribution story” (i.e., the 
narrative description of the theory of change and its supporting evidence). 


Figure 4. Contribution Analysis Process 


Step 1: Set out the cause-effect issue to be addressed 

Step 2: Develop the theory of change 

Step 3: Assess the resulting contribution story 

Step 4: Gather the existing evidence on the theory of change 
Step 5: Reassess the contribution story and challenges to it 
Step 6: Seek out additional empirical evidence 

Step 7: Revise and strengthen the contribution story 


It should be noted that Steps 1 to 5 are best undertaken during the design phase of the 
intervention (see Section 10.0), rather than at the evaluation planning stage. These first five steps 
comprise a contribution analysis framework. 


The seven steps are presented and discussed in sequential order, but as noted below, these steps 
will usually be subject to constant review and revision. 


Step 1: Set Out the Cause-Effect Issue to Be Addressed 
The first step in a contribution analysis is to clarify the scope of the evaluation and articulate the 


evaluation questions to be addressed: 


> Articulate clearly what cause-effect issue is being addressed, usually in the form of questions 
such as: 


— Has the intervention made a contribution to addressing the problem? 


— What aspects of the intervention or the context led to a contribution being made? 
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> Considering the evaluation context and the nature of the decisions to be informed by the 
evaluation, determine the level of confidence required for the findings. For lower-risk 
interventions, the confidence level requirements are generally lower; accordingly, the scope of 
evidence and the depth of analysis required are generally lower. 


> Explore the nature and extent of the contribution expected from the intervention. Determine 
what evidence would help to confirm that the intervention made a noticeable contribution to 
the intended results. 


> Assess whether the expected contribution is generally plausible given the nature of the 
intervention (i.e., determine whether the problem can reasonably be addressed through the 
intervention). If a contribution is not plausible, the value of further analysis may be limited. 


Step 2: Develop the Theory of Change 


Developing a theory of change is the second step in a contribution analysis: 
> Build an initial and a refined theory of change for the intervention (see Section 6.0). 


> In building the theory of change, determine the level of detail needed. Contribution analysis is 
best done with reasonably straightforward and not overly detailed results chains, especially at 
the outset. At later steps, it may be decided that further detail and refinements are required to 
further explore some aspects of the theory of change, but these can be added later. 


Step 3: Assess the Resulting Contribution Story 


At this point it is useful to critically review the contribution story resulting from the developed 
theory of change: 


> Assess the logic of the links and test the plausibility of the assumptions in the theory of 
change: Are there significant gaps in the theory? Can they be filled by further refining the 
theory of change? If not, what other steps are needed to produce a credible evaluation? 


> Identify where evidence is required to strengthen the contribution story: Which links are 
supported by minimal or no evidence? Which external factors are not well understood? 


> Determine how much the theory of change is contested: Is it widely agreed to? Are specific 
aspects contested? Are there several theories of change at play? 


Some of this thinking and analysis will have started with the development of the theory of 
change at Step 2. Accordingly, there will often be a process of iteration between these two steps. 
An independent and critical review of the theory of change (e.g., by an external expert) may be 
useful at this point in the process. The theory of change at this stage is the prior-hypothesized 
theory of how the designers of the intervention expected it work. It is now ready to be tested 
against the evidence. 
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Step 4: Gather Existing Evidence on the Theory of Change 


Before gathering new data, it is useful and cost-effective to look at relevant existing data and 
information related to the theory of change: 


> Gather existing evidence, such as prior evaluations and research, ongoing monitoring and 
environmental scans, and program reports, to provide empirical evidence for the contribution 
story that the intervention is claiming (e.g., evidence on activities implemented, observed 
results, assumptions, relevant external factors, and manifestations of risk). 


> Gather evidence on the identified external influences (i.e., data that help determine the 
plausibility of these influences on the theory of change). In some cases, evaluators may want 
to develop theories of how the external influences may be affecting the program. 


At this stage in the analysis, a theory of change for the intervention has been developed and the 
available evidence supporting the theory of change has been gathered. The theory of change has, 
to some extent, been tested. As well, the significant external factors have been identified and any 
supporting evidence has been gathered. 


Step 5: Reassess the Contribution Story and Challenges to It 


The contribution story arising from the theory of change can now be critically assessed in light of 
the existing evidence: 


> To critically assess the contribution story: 


— Determine which links in the theory of change are strong (strong logic; good evidence 
available that the assumptions held; low risk and/or wide acceptance), and which are 
weak. 


— Assess the overall credibility of the story. Does the pattern of observed results and links 
validate the results chain? 


— Determine whether the stakeholders agree with the contribution story developed. 


— Assess the likelihood that any of the significant external factors have had a noteworthy 
influence on the observed results. 


— Identify the main weaknesses in the story. For example, links in the theory of change 
could be rated on the likelihood of their being realized.’ Any weaknesses point to areas 
where additional data or information would be useful. 


3. An example of such rating of links in a theory-based evaluation can be found in Review of outcomes to impact 
(RO): Practitioner's handbook (Global Environment Facility Evaluation Office, 2009). The authors rate from both 
a theoretical perspective and a delivery perspective. 
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> Adjust the theory of change, disaggregating elements if needed, and reassess the contribution 
story. 


Thus far in the evaluation process, no new data have been gathered. Rather, the assessment 
process has relied on existing intervention documents, discussions with management and 
possibly experts, and a literature review. Step 5 identifies where additional evidence from 
evaluation is needed to support the contribution story. This evidence can serve as a key driver for 
the development/redevelopment of the evaluation framework and updates to the related data 
collection methods. This evaluation framework should include the five core issues from the 
Directive on the Evaluation Function “"' as well as additional issues identified by evaluation users 
and during Steps 1 to 5 outlined above. 


Step 6: Seek Out Additional Empirical Evidence 


It is not until this step that the collection of new primary data for the evaluation begins, informed 
by the previous steps. At this stage: 


> Evaluators should gather the evidence needed to strengthen the contribution story, using 
appropriate data gathering techniques discussed in Program evaluation methods: 
Measurement and attribution of program results" (e.g., surveys, interviews, reviews and 
analyses of administrative data). The goal is to seek data that provide evidence of results 
occurring; on the validity of the assumptions and risks in the theory of change; and about 
significant external factors that may have influenced the results achieved. 


> There may be quasi- or experimental designs involving comparison groups that could be used 
to explore elements of the theory of change. Weitzman, Silver and Dillman (2002) discuss an 
example of strengthening a theory of change approach using a quasi-experimental comparison 
group. 

> From a theory-based perspective, several standard data-gathering techniques can be used to 
strengthen the theory testing: 


— Key informant interviews can be used to test the theory of change, to elicit alternative 
theories of change which the key informants might have, and to discuss other influencing 
factors. When collecting responses from interviewees, it is helpful to clarify the evidence 
on which they are basing their views. 


— Focus groups and workshops are good methods to explore a theory of change, since these 
allow for discussion and debate about how different stakeholders see the intervention 
working. Alternative theories of change may emerge and other influencing factors may 
be identified. These can be used as a means to develop, or further develop, a theory of 
change, and as a way to determine and identify evidence on the extent to which the 
theory of change was realized in practice. 
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— Case studies can be used in the same way as focus groups and workshops. If put in the 
context of a theory of change, case studies are more powerful as a data-gathering tool in 
helping to confirm or refute a theory of change, or the “micro steps” in a theory of 
change, showing that the theory of change is indeed plausible and not just based on 
unsupported beliefs. 


Step 7: Revise and Strengthen the Contribution Story 


Using the new evidence gathered above, evaluators can now build a more credible contribution 
story, with strengthened conclusions on the causal links in the theory of change. It bears 
repeating that theory-based approaches such as contribution analysis work best as an iterative 
process. Accordingly, at this point in the analysis, the evaluator may see a need to return to 

Step 5 or even earlier steps to reassess the strengths and weaknesses of the theory of change and 
the contribution story, and to decide whether further analysis would be useful or possible. 
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8. 


Strengths and Weaknesses of Theory-Based 
Approaches to Evaluation 


Theory-based approaches to evaluation are not a panacea for attributing results to programs. 
They do, however, present a number of positive features: 


> 


They can often be undertaken in circumstances where other approaches (e.g., experimental 
designs) cannot be used. 


They allow evaluators and intervention managers to tell a contribution story that makes sense 
to those involved. 


They open the black box of the intervention, allowing evaluators to arrive at findings on why 
interventions are working or are not working. 


They allow conclusions to be drawn on the cause-effect elements of an intervention. 


They can help leverage existing data to a greater extent and help focus new data collection on 
areas where there are significant gaps, resulting in more efficient and effective use of 
evaluation resources. 


At the same time, there are clear challenges in using theory-based approaches: 


> 


They do not necessarily provide a quantitative measure of the size of the contribution an 
intervention is making. If this is required, there may still be a need for analysis that supports 
measurement of the size of observed results. 


Developing a theory of change can be difficult because it involves synthesizing a range of 
views and information sources as well as obtaining the agreement of stakeholders. 


In some situations, developing a theory of change can be time-consuming and/or require 
significant amounts of data. However, in some cases (e.g., low-risk programs or 
low-complexity programs where the tolerance for uncertainty in attribution is higher), there 
may be an opportunity for using a calibrated “lighter-touch approach” (e.g., using a 
less-detailed theory of change with less testing). In doing so, evaluators may be able to add 
rigour to, and enhance the credibility of, these evaluations (including evaluations involving 
small sample sizes (White and Phillips, 2011) or incorporate expert opinion as part of their 
methodology). 


More than one theory of change may emerge. If multiple theories of change emerge and are 
strongly held, they may have to be tested against the evidence to see which theory best 
reflects reality. In such cases, evaluators may want to focus their efforts on where the theories 
differ, exploring the reasons for, and implications of, this difference. 


Readers are referred to Weiss (1997) and Mackenzie and Blamey (2005) for more detailed 
discussions about the challenges of using theory-based evaluations. 
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When introducing theory-based evaluation approaches to their departmental evaluation tool box, 
federal evaluators may wish to start small, with relatively simple interventions, and build up 
experience, helping to overcome some of the challenges described above. The references in this 
document provide a number of examples and discussions of using theory-based approaches. 
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9, Using Theory-Based Approaches to Evaluate 
Cause-Effect Issues in Different Types of 
Interventions 


The federal evaluation community is responsible for and continually seeks innovations in how it 
evaluates various types of interventions. These include evaluating policies, horizontal initiatives, 
low-risk programs, grants and contributions, and cluster groupings of interventions. This section 
discusses how theory-based approaches can be applied to these endeavours. In some cases, such 
as policy evaluation, there is experience to build on. In other cases, suggestions are made that 
need to be explored in practice. 


Evaluating Policies 


Evaluating government policies can be challenging. Government policies typically comprise a 
number of different program-level interventions (or other activities) for realizing the objectives 
of the policy. These program interventions may be of quite different types, reflecting the use of 
different policy instruments, such as direct spending, legislation, regulation or taxation, and the 
specific types of activities undertaken within these broad groups. There may also be interactions 
among the various program interventions, or between policies, which increase the challenge of 
evaluating such policies. 


The application of theory-based approaches in these cases could take a number of forms, 
including but not limited to the following: 


> Develop a theory of change for the policy, outlining how it is expected to work, showing the 
suite of supporting program interventions and how they are expected to work in concert to 
achieve the expected results of the policy. This theory can then provide a basis for deciding 
how to go about evaluating the policy. 


> Develop theories of change for each program intervention, aimed at assessing the contribution 
being made to the overall policy (e.g., a contribution analysis). The findings from the 
individual evaluations can then be aggregated and synthesized to arrive at findings and 
conclusions about the policy. In the individual theories of change developed for each program 
intervention, the other related program interventions (i.e., the other program interventions 
supporting the policy) can be treated as external factors. Accordingly, an indication of the 
interactions among the interventions can be assessed. 
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> Rather than focusing on program interventions, it may be useful to identify the different types 
of strategies” being used to implement the policy, and use these intervention strategies as the 
focus of the theory-based evaluation. This perspective seeks to answer questions such as: 
Which policy mechanisms work well? For whom? In what circumstances? This perspective 
becomes useful when the various program interventions use different policy mechanisms. If 
each program intervention uses a different policy mechanism, each program intervention 
could be evaluated separately, as discussed above. However, in such cases, an analysis of 
policy cohesion may be needed to provide an integrated picture of the impact of the policy 
instrument. 


In all cases, it should be remembered that theories of change are abstractions aimed at 
developing an overview of how an intervention works. In certain circumstances, especially in 
more complex settings, it may also be useful to undertake detailed case studies to test theories, 
aspects of theories or the multiple theories of change that have been developed. 


Stame (2004) and Vaessen and Todd (2007, 2008) discuss an approach to evaluating policy-type 
interventions. 
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Horizontal Initiatives 


From a methodological perspective, horizontal initiatives are quite similar to the policy 
evaluation case described above; the horizontal initiative can be seen as the overarching policy 


4. Generic policy strategies are often grouped into three categories: incentives (i.e., rewards for behaving in a 
certain manner), deterrents (i.e., sanctions for not behaving in a certain manner), and appeals (i.e., calls to 
behave in a certain manner based on it “being the right thing to do”). (Bemelmans-Videc et al., 1998) 
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and the individual program components as the equivalent of the program interventions being 
implemented to fulfill the policy goals. Horizontal initiatives have an additional challenge, in 
that their component interventions are being managed by different departments. Horizontal 
initiatives thus require significant governance and coordination of activities, including program 
management and performance measurement. This need for governance and coordination extends 
to the planning and undertaking of evaluations of horizontal initiatives. 


Similar to the policy evaluation case, a theory-based approach to evaluating a horizontal 
initiative may focus on the overarching intervention theory (i.e., the horizontal theory); the 
theories of the program interventions being undertaken by partner departments (i.e., the different 
departmental strategies being used); or ideally both. The latter allows for a contribution analysis 
based on the interconnected theories of change. Also similar to the theory-based approach to 
policy evaluation discussed above, the theorized interactions among the interventions (i.e., the 
interventions in other departments) could be seen as possible external factors affecting the 
theories of change at work in other parts of the horizontal initiative. At the same time, specific 
interventions may also contribute to other theories of change within a department. 


In certain types of horizontal initiatives, departments may use different intervention mechanisms 
as part of the collective effort. This may provide a natural experiment for learning about and 
comparing experiences using theory-based approaches. 


Low-Risk Interventions 


In a low-risk intervention, the combination of risks (e.g., low materiality, low impact of program 
failure, evidence from a prior evaluation suggesting good program performance, no significant 
questions about program relevance, and other factors [e.g., contextual stability]) suggests that an 
in-depth evaluation is not needed at this time, or that the evaluation should put greater emphasis 
on evaluating specific program elements and less on others. In these cases, the evaluation 
approach, scope, design and methods can be calibrated relative to the required confidence level. 
A “light-touch” theory-driven approach may allow the evaluation team to make causal inferences 
at a higher level of rigour than other approaches, while still using limited resources. This might 
entail: 


> using a simplified results chain with a basic theory of change; 
> identifying a few critical assumptions and risks; 


> confirming the reasonableness of the intervention design (i.e., the theory of change makes 
sense); 


> confirming that the planned activities of the intervention were carried out, resulting in the 
planned outputs; 
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> 


if monitoring data are available, confirming that at least some of the immediate and 
intermediate outcomes occurred; 


undertaking selected interviews of key stakeholders to confirm that the theory of change is 
working as intended and that other influencing factors did not play a major role; and 


drawing conclusions on the extent to which the program activities are making a difference. 


Depending on the case, any of these steps could be enhanced to provide a greater level of 
confidence in the conclusions reached. The design would be tailored to the perceived risk and 
level of confidence needed. 


Grants and Contributions Programs 


The evaluation of grants and contributions programs (Gs & Cs) presents a number of challenges 
that a theory-based approach can help address. These include: 


> 


the limited capacity of organizations receiving the funds to gather data and assess their own 
services or projects; 
operating contexts, or limits on capacities of fund receiving organizations, that prohibit the 


use of experimental or quasi-experimental evaluation designs; and 


data collection from recipients that sometimes results in positive feedback only (“yes, things 
are working well”), on the assumption that this will help ensure the approval of their next 
funding application. 


In many Gs & Cs cases, it may be useful to develop two theories of change: one for the 
department and one for the recipient organization(s). The departmental theory of change would 
involve the assumptions and beliefs of why, with additional funds and perhaps additional 
capacity building, the recipient organization would be able to deliver and report on the expected 
enhanced services and benefits for Canadians. The recipient’s theory of change is the one behind 
the delivery of services to the intended beneficiaries (i.e., why the recipient believes that by 
providing the expanded services, the beneficiaries will be helped). These two theories of change 
would provide a framework to identify, clarify and separate the department’s direct interests and 
accountabilities from those of the recipient. 


A recipient theory of change may also provide the basis for the recipient to tell its performance 
story in a credible manner. This may act as an incentive for the recipient to gather evidence for, 
and improve the telling of, their performance story. This could include gathering “Most 
Significant Change” stories (Dart & Davies 2003) from their clientele to confirm the theory of 
change and to identify key contextual factors and mechanisms of change that will help them 
deliver better services. 
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A theory-based evaluation could provide a means for the department to make more credible 
claims about the difference its funds are making by verifying the theories of change at play. In 
addition, with a theory of change as a framework, interviews, focus groups and case studies (see 
Section 7.0) could be used to provide better information on the robustness of the theories. 
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Evaluating Clusters or Portfolios of Interventions 


Under the Policy on Evaluation, departments and agencies are required to evaluate their direct 
program spending (DPS) at least once every five years. As noted in A guide to developing 
departmental evaluation plans (Treasury Board of Canada Secretariat, 2010), departments can 
choose their units of evaluation based on configurations that allow them to both (a) undertake 
evaluations of reasonable units of effect that meet the information needs of evaluation users and 
b) meet their legal and policy obligations. Thus, some departments choose to undertake 
evaluations that incorporate several program interventions. This sometimes involves undertaking 
evaluations of multiple lowest-level elements of the Program Alignment Architecture (PAA), 
usually at the subactivity or sub-subactivity level. In other cases, this might involve a portfolio of 
program subcomponents (often Treasury Board submission-defined programs) found within a 
single lowest-level PAA element (e.g., a sub-subactivity). These are commonly referred to as 
cluster or portfolio evaluations. 


In some cases, departments may opt to evaluate each of the lowest level PAA elements 
separately and then form general conclusions about the higher-level Program Activity. 

A theory-based evaluation of a cluster or portfolio of interventions may allow departments to 
assess, compare and contrast the contributions of the interventions and thus arrive at a more 
holistic picture of the performance story for the lowest-level PAA programs. Similar to the 
policy and horizontal evaluation initiatives referred to above, these evaluations may benefit from 
developing both program-level and subprogram-level theories of change and testing these as part 
of the evaluation process. One of the challenges in undertaking cluster or portfolio evaluations 
however is the need to have alignment in the performance measurement strategies across 
program interventions. 


Alternatively, a program-activity-level theory of change could be helpful in identifying themes 
that run through the various subactivities or sub-subactivities, which in turn could prove useful 
for an evaluation. Indeed, a performance measurement strategy and a multi-year evaluation plan 
could be developed at the outset for the cluster/portfolio (e.g., a program activity). Over a period 
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of time, the evaluation of each program subactivity or sub-subactivity could be undertaken using 
the common framework and theory of change developed. 


Further, in cases where the program subactivities or sub-subactivities are similar, economies 
might be realized by using a common framework for the evaluation. In a theory-based approach, 
this could mean developing one generic theory of change for the cluster/portfolio and using it to 
identify what performance measurement and evaluation data are needed. For evaluation data, this 
would include developing common interview guides and data collection tools, as well as 
deciding what data should be collected for which context and for which external factors. For 
each of the interventions evaluated in the cluster/portfolio, data could then be collected and 
analyzed. À model similar to this approach is discussed by Barley and Jennes (1993). 


As well, if the generic theory of change is tested against the various program subactivities or 
sub-subactivities in terms of its plausibility and logic, any program subactivities or 
sub-subactivities that do not fit well into the overall program activity can be identified prior to 
any data collection, possibly leading to a restructuring of the PAA. 


In addition to having evaluations for specific interventions in the cluster/portfolio, data and 
analysis would be available to compare across the cluster and to make more general findings on 
the circumstances in which this type of intervention works well. A theory-based perspective is 
especially suited for this type of analysis and facilitates learning in the cluster. 
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10. Other Uses of Theory-Based Approaches 


As previously suggested, theories of change can do more than facilitate an evaluation. In 
particular, program theories and theories of change can be used by: 


> intervention designers for planning and designing interventions; 
> intervention managers for designing performance measurement systems for interventions; and 


> those responsible for reporting, to provide a framework for reporting on the performance of an 
intervention. 


Planning and Designing Interventions 


It is common for evaluators to find that a logic model, results chains, and theory of change have 
not been developed or are only partially developed for the intervention to be evaluated. 
Developing a basic or more detailed theory of change as part of the upfront intervention design, 
perhaps with help from evaluators, provides a number of benefits, including: 


> providing a means to reach agreement among stakeholders of just how the intervention is 
expected to contribute to its intended aims. In more complicated situations, it can help clarify 
the intended contribution by the various subcomponents of a broader intervention; 


> identifying aspects of the intervention design that need attention, in particular, ensuring that 
the key assumptions required for the intervention to work can be directly or indirectly 
influenced by the intervention; 


> identifying where research may be needed to better understand how the intervention works; 
and 


> communicating to others what the intervention is intending to achieve and how it will produce 
that result. 


Designing Performance Measurement Strategies 


The theory of change helps identify not only which results should be monitored, but also which 
other factors should be followed so that the intervention can be kept on track and progress 
monitored. A review of the theory of change can help identify which results, which assumptions, 
which risks and which external factors would be most useful for the manager to monitor through 
performance measures. Properly introduced, theories of change ought to be attractive to 
managers. They contribute greatly to effective managing for results (Mayne 2009). Finally, if 
good monitoring data are available, stronger and more efficient evaluations can be undertaken. 
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Reporting on performance 


The theory of change provides a framework for the intervention, and collectively the department, 
to tell its performance story. Data and information that are collected can be reported against the 
theory of change to help show that the intervention is making a difference. The framework also 
provides a rational and transparent means for selective reporting on performance. 
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11. From Concept to Practice 


How a theory-based evaluation is carried out will vary from case to case. The material in this 
document stresses that theory-based approaches are ways of thinking about and designing 
approaches to address cause-effect issues in evaluations. This guide should not be considered as 
a how-to manual but as a set of principles and guidelines on how to make use of theory-based 
approaches. 


Theory-based approaches were initially developed to help in the evaluation of more complicated 
and complex interventions. Much of the literature discusses this experience. Clearly, these 
approaches also apply to the more straightforward evaluation cases, being particularly useful 
when experimental designs are not practical. They can help not only to strengthen otherwise 
weak evaluation designs but also to provide evidence on the perennial evaluation question of 
attribution: Has the intervention made a difference? 


The challenge to federal evaluators is to build a base of practical experience in using 
theory-based approaches. 
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12. For More Information 


For more information on evaluations and related topics, please visit the Evaluation” section of 
the Treasury Board of Canada Secretariat website. 


Alternatively, please contact: 


Centre of Excellence for Evaluation 
Expenditure Management Division 
Treasury Board of Canada Secretariat 
Email: evaluation@tbs-sct.gc.ca 
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